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ABSTRACT 

Many  military  planning  problems  are  difficult  to  solve  us¬ 
ing  pure  mathematical  programming  techniques.  One  such 
problem  is  scheduling  unmanned  aerial  vehicles  (UAVs)  in 
military  operations  subject  to  dynamic  movement  and  con¬ 
trol  constraints.  This  problem  is  instead  formulated  as  a 
dynamic  programming  problem  whose  approximate  solu¬ 
tion  is  obtained  via  the  Assignment  Scheduling  Capability 
for  UAVs  (ASC-U)  model  using  concepts  from  both  simu¬ 
lation  and  optimization.  Optimization  is  very  effective  at 
identifying  the  best  decision  for  static  problems,  but  is 
weaker  in  identifying  the  best  decision  in  dynamic  sys¬ 
tems.  Simulation  is  very  effective  in  modeling  and  captur¬ 
ing  dynamic  effects,  but  is  weak  in  optimizing  from  alter¬ 
natives.  ASC-U  exploits  the  relative  strengths  of  both 
methodologies  by  periodically  re-optimizing  UAV  assign¬ 
ments  and  then  having  the  simulation  transition  the  states 
according  to  state  dynamics.  ASC-U  thus  exploits  the 
strengths  of  simulation  and  optimization  to  construct  good, 
timely  solutions  that  neither  optimization  nor  simulation 
could  achieve  alone. 

1  INTRODUCTION 

Military  operations  are  dynamic,  complex  series  of  events. 
The  problems  associated  with  military  operations  are  natu¬ 
rally  also  dynamic  and  complex.  This  paper  describes  the 
dynamic  problem  associated  with  the  allocation  of  UAVs 
from  their  owning  military  units  to  mission  areas  over  time 
in  Section  2.  The  complex  dynamics  of  the  problem  are 
evident  from  ground  control  stations  and  the  launch  and 
recovery  sites,  from  which  the  UAVs  are  launched,  return, 
and  are  recovered,  each  of  which  move  from  one  time- 
stamped  waypoint  to  the  next  throughout  the  scenario.  The 
use  of  fuel  during  flight  and  the  recovery  time  once  at  a 
launch  and  recovery  site  are  sources  of  additional  dynam¬ 
ics. 


In  Section  3,  the  paper  describes  why  classical 
mathematical  programming  or  simulation  alone  have  diffi¬ 
culties  solving  this  highly  dynamical  scheduling  problem. 
The  strengths  and  weaknesses  of  the  two  approaches  are 
discussed.  The  Dynamic  Allocation  of  Fires  and  Sensors 
simulation,  on  which  ASC-U  is  built,  is  described  in  Sec¬ 
tion  4. 

A  dynamic  programming  formulation  is  explained  in 
Section  5.  This  section  explains  how  a  dynamic  program¬ 
ming  approximation  can  exploit  the  special  structure  of  the 
problem  to  obtain  quality  solutions  quickly.  Then,  a  solu¬ 
tion  technique  that  makes  use  of  the  strengths  of  a  com¬ 
bined  simulation  and  optimization  approach  is  described  in 
Section  6,  and,  finally,  initial  results,  insights,  and  conclu¬ 
sions  are  discussed  in  Section  7. 

2  THE  UNMANNED  AERIAL  VEHICLE 

SCHEDULING  PROBLEM 

We  consider  a  dynamic  unmanned  aerial  vehicle  routing 
problem.  In  addition  to  unmanned  aerial  vehicles  (UAVs), 
this  problem  accounts  for  moving  launch  and  recovery 
sites  (LRSs)  from  which  UAVs  are  launched  and  must  re¬ 
turn,  moving  ground  control  stations  (GCSs)  from  which 
UAVs  must  be  controlled  at  every  instant  during  flight  and 
that  can  control  a  finite  number  of  UAVs  at  any  one  time, 
and  stationary  missions  from  which  a  time-dependent 
value  is  received  if  a  UAV  with  the  appropriate  sensor  vis¬ 
its.  The  objective  is  to  determine  the  maximum  sum  of 
mission  values  that  can  be  accomplished  given  the  number 
of  each  type  of  UAVs  and  their  performance  characteristics 
of  flight  time  and  speed. 

Missions  are  characterized  by  a  set  of  sensor  require¬ 
ments  at  a  Cartesian  coordinate.  Each  sensor  requirement 
has  a  start  and  end  time,  a  type  of  sensor  required  and  a 
value  rate.  When  a  UAV  with  the  correct  sensor  arrives  at 
the  same  location  as  a  sensor  requirement,  and  that  re¬ 
quirement  does  not  yet  have  a  sensor  accomplishing  that 
requirement,  the  UAV  receives  credit  for  accomplishing 
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that  sensor  requirement.  The  value  received  is  the  UAV’s 
time-on-station  multiplied  by  the  value  rate  of  the  sensor 
requirement  that  is  satisfied.  The  UAV  receives  value  until 
the  UAV  departs  or  the  sensor  requirement  ends.  Since 
UAVs  carry  sensor  packages  with  potentially  many  sen¬ 
sors,  multiple  missions  may  be  satisfied  and  these  values 
are  additive. 

UAVs  are  characterized  by  a  unique  name,  a  type,  a 
starting  and  ending  LRS,  an  operating  time,  a  speed,  a 
transition  time  for  recovery  and  refueling,  an  allowable  op¬ 
erating  radius  from  a  controlling  GCS,  a  sensor  package, 
and  an  available  start  time  and  an  end  time.  The  unique 
name  identifies  each  UAV  in  order  to  capture  which  sensor 
requirements  it  accomplished  and  its  state  throughout  the 
problem.  The  type  of  UAV  is  important  to  ensure  that  only 
GCSs  and  LRSs  that  are  compatible  with  a  given  UAV  are 
used  for  control  and  recovery,  respectively.  Starting  and 
ending  LRS  designations  for  each  UAV  allow  for  the  ini¬ 
tial  and  terminal  conditions  of  the  problem  and  support  the 
requirement  that  a  UAV  should  begin  and  end  at  the  ap¬ 
propriate  organizational  unit.  Operating  time  and  speed 
dictate  whether  a  mission  is  feasible  for  a  given  UAV  and 
how  much  time-on-station  the  UAV  can  achieve  before  it 
must  return  to  an  LRS  for  recovery  and  refueling.  The 
transition  time  for  recovery  and  refueling  provides  the  time 
a  given  UAV  must  spend  at  an  LRS  before  it  can  be 
launched  again.  The  allowable  operating  radius  is  the 
maximum  distance  a  UAV  can  be  from  a  GCS  and  still  be 
controlled  by  that  GCS.  A  sensor  package  is  a  set  of  sen¬ 
sors  that  can  be  assigned  to  a  UAV  for  a  particular  flight. 
The  available  start  and  end  times  for  a  UAV  determine  the 
times  it  should  be  considered  as  available  to  fly. 

Sensor  packages  are  sets  of  sensors,  weapons,  or 
communications  capabilities.  A  sensor  package  is  assigned 
to  a  UAV  and  determines  what  sensor  requirements  that 
the  UAV  can  accomplish.  Value  is  only  gained  for  a  sensor 
requirement  if  the  UAV’s  sensor  package  has  the  appropri¬ 
ate  sensor  in  its  set. 

LRSs  are  characterized  by  a  unique  name,  a  UAV 
type,  a  Cartesian  coordinate,  and  a  capacity.  The  LRS  is 
where  the  UAV  must  originate  and  end  its  flight.  The  LRS 
is  also  where  the  UAV  will,  upon  landing,  transition 
through  recovery  and  refueling  before  becoming  available 
to  depart  to  accomplish  more  missions.  LRSs  may  move 
over  time. 

GCSs  are  also  characterized  by  a  unique  name,  a  UAV 
type,  a  Cartesian  coordinate,  and  a  capacity.  The  GCS  con¬ 
trols  the  UAV  during  flight  and  may  provide  the  commu¬ 
nications  link  over  which  its  information  is  passed  when 
necessary.  GCSs  may  move  over  time. 


3  MATHEMATICAL  PROGRAMMING  VS. 
SIMULATION 

In  a  mathematical  programming  problem,  one  seeks  to 
minimize  or  maximize  a  real  function  of  real  or  integer 
variables,  subject  to  constraints  on  the  variables.  An  inte¬ 
ger  programming  problem  takes  the  form: 


subject  to 


ma  xcTx 

Ax  =  b ,  where  igZ 


where  c  is  the  vector  of  costs  or  rewards,  x  is  the  decision 
variable,  and  Z  is  the  set  of  integers.  Using  this  formulation 
to  model  the  UAV  scheduling  problem  has  several  pitfalls. 
If  uniform  time  steps  are  used  for  travel  to  and  from  the 
missions  the  solution  to  an  integer  program  may  not  be 
feasible.  Additionally,  the  dynamics  involved  are  difficult 
to  model  in  the  constraint  function,  Ax-b  . 

Even  with  effective  modeling,  the  dynamics  and  time- 
steps  of  the  problem  cause  the  number  of  decision  vari¬ 
ables  to  suffer  from  high  dimensionality  so  that  extensive 
decomposition  techniques  may  be  needed  to  solve  the 
problem  to  optimality  or  to  even  find  a  near-optimal  feasi¬ 
ble  solution.  Solution  times  for  large  integer  problems  are 
notoriously  long. 

Mathematical  programming  is  a  powerful  tool  for 
more  static  problems.  However,  for  dynamic  problems  it 
suffers  from  an  exponential  number  of  variables  and  con¬ 
siderable  solution  challenges. 

On  the  other  hand,  simulation  allows  straightforward 
and  efficient  modeling  of  dynamics,  especially  of  moving 
entities  (Buss  and  Sanchez  2005).  Although  simulation  al¬ 
lows  movement,  representation  of  state  changes  and,  easy 
collection  of  statistical  measures  of,  by  itself  it  does  not 
provide  the  decision  capacity  needed  to  solve  effectively 
tell  UAVs  where  to  go  next.  Myopic  decision  policies  are 
often  used  within  a  simulation  but  these  policies  would  be 
inadequate  for  the  UAV  scheduling  problem  which  re¬ 
quires  sequential  decisions  over  time  and  space  in  a  GCS 
capacity  constrained  environment. 

The  need  for  complex  decisions  prohibits  the  use  of  a 
traditional  simulation  to  determine  the  effects  of  a  given 
allocation  of  UAV  systems.  The  complexities  of  the  prob¬ 
lem  caused  by  moving  LRSs  and  GCSs  and  their  capacities 
prohibit  the  problem  from  being  easily  formulated  using 
mathematical  programming  formulations  which  are  better 
suited  for  stationary  problems.  Instead,  the  problem  is 
viewed  and  formulated  as  a  dynamic  programming  prob¬ 
lem  and,  furthermore,  solved  using  approximate  dynamic 
programming  techniques  using  a  combination  of  simula¬ 
tion,  to  model  movement,  and  simple  mathematical  pro¬ 
gramming,  to  model  the  allocation  decisions. 
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4  DYNAMIC  ALLOCATION  OF  FIRES  AND 
SENSORS  SIMULATION 

The  Dynamic  Allocation  of  Fires  and  Sensors  (DAFS) 
simulation  is  a  low-resolution,  constructive  entity-level 
simulation  framework  designed  for  combat  modeling  (Ha¬ 
vens  2002;  Buss,  Ahner,  and  Ruck  2006).  The  DAFS 
framework  consists  of  a  Discrete  Event  Simulation  Model 
with  embedded  optimization,  Extensible  Mark-up  Lan¬ 
guage  (XML)  input  and  output  modules,  and  an  output 
analysis  package  (Ahner  2005).  Through  the  use  of  an  in¬ 
terchangeable  component-based  architecture,  the  simula¬ 
tion  provides  the  user  extensive  ability  to  modify  entities, 
configurations,  simulation  parameters  and  data  output. 
DAFS,  using  a  low  resolution  approach,  runs  fast  and  is 
easy  to  set  up. 

DAFS  has  a  component-based  architecture.  Within  the 
simulation  model,  there  are  two  types  of  components  that 
work  together  to  give  DAFS  its  overall  capability,  physical 
components  and  functional  components.  Physical  compo¬ 
nents  represent  physical  items  such  as  sensors  and  muni¬ 
tions.  Functional  components  control  some  action  within 
the  simulation.  One  functional  component  is  the  con¬ 
strained  value  optimizer  (CVO). 

The  Constrained  Value  Optimizer  (CVO)  is  the  com¬ 
ponent  that  provides  a  myopic  optimal  solution  for  the  cur¬ 
rent  state  of  the  simulation.  The  CVO  allows  for  the  reallo¬ 
cation  of  BLUE  platform  resources  to  improve  the  BLUE 
forces  near-term  chances  of  successfully  completing  its 
mission. 

The  CVO  does  not  perform  global  dynamic  optimiza¬ 
tion  which  would  be  computationally  intractable  for  large 
problems.  Instead,  the  CVO  takes  the  current  state  of 
BLUE  resources,  compares  the  capabilities  of  BLUE 
forces  to  the  state  of  demands,  and  through  its  mathemati¬ 
cal  formulation  of  the  problem  determines  the  ‘optimal’ 
actions  to  take  according  to  the  current  states  modeled  in 
the  mathematical  programming  formulation.  Since  this  ap¬ 
proach  has  a  control  theory  flavor  to  it,  it  has  been  called 
model  predictive  control.  The  CVO  can  employ  either  a 
mathematical  programming  solver  or  a  heuristic. 

As  currently  implemented,  the  objective  function 
formulated  for  mathematical  programs  is  solved  using 
LPSolve  5.5  (LpSolve  2005).  LPSolve  5.5  is  an  im¬ 
plementation  of  the  simplex  method  for  linear  program 
formulations  and  integer  program  formulations  using 
branch  and  bound.  LP_  Solve  is  implemented  in  the  C  pro¬ 
gramming  language,  so  DAFS  communicates  with  it  using 
the  Java  Native  Interface  (JNI). 

In  DAFS,  the  constrained  value  optimizer  is  imple¬ 
mented  as  an  interface.  A  CVO  is  constructed  and  is  called 
periodically  to  cue  UAV  allocation  decisions  given  the  cur¬ 
rent  state  of  the  simulation. 

This  use  of  a  control,  given  a  current  state,  falls  within 
the  field  of  dynamic  programming,  a  methodology  for  op¬ 


timization  of  sequential  decisions.  The  Dynamic  Pro¬ 
gramming  formulation  of  the  UAV  assignment  problem  is 
discussed  in  the  next  section. 

5  DYNAMIC  PROGRAMMING  FORMULATION 

Consider  a  finite  state  and  discrete  time  horizon  dynamic 
programming  problem.  Let  S  be  the  state  space  of  the  sys¬ 
tem.  The  finite  time  horizon  is  t  =  0,...,T  where  the  time 
steps  may  or  may  not  be  uniform.  The  state  St  e  S  repre¬ 
sents  the  state  at  time  t  =  0 ,...,T  .  A  decision  vector  ut  that 
acts  on  the  system  is  selected  from  a  finite  set  U  at  each 
time  step.  The  state  evolves  according  to  a  state  equation 
which  has  the  form 

S t+T  ~  f\  (*-h  ?  ut  ? 

where  fx  is  a  function  describing  the  systems  dynamics, 

St  includes  the  state  of  the  missions,  LRSs,  GCSs,  and 

UAVs  which  are  defined  in  the  next  section,  and  r  is  the 
time  to  the  next  decision  event. 

A  policy  is  a  mapping  H(St) :  St  -» t/that  determines  a 

decision  as  a  function  of  the  state,  i.e.  ut  =  n(N) .  We  rep¬ 
resent  the  one  period  contribution  to  the  reward  as 
Ct(St,ut).  We  express  the  T  -stage  value  to  be  maximized 
as  the  value  of  the  summation  of  the  T  costs: 

t 

max  YCf(Sf,uf) 

uteU(St)t= o 

The  cost-to-go  or  future  value  function  at  time  t  is 

Jt(St)=  max  TCt(St,ut) 

ut,...,uT  T=t 

s-t.  St+T  =  fx{St,ut,z)  for  t  =  0,...,T - 1 
We  can  solve  for  Jt{St)  using  the  recursive  equation: 

Jt(St)  =  maxCt(St,ut)  +  Jt+T(St+T)for  t  = 

utsU 

It  is  in  this  context  of  dynamic  programming  that  we  de¬ 
scribe  our  formulation  and  methodology. 

5.1  State  Space  Description 

The  state  St  eS  is  defined  as 

5,=  |  u  sYAV<,  u  s?CSj,  u  sRRSk,  u  S™ssl°n‘\ 

[isUAV  jsGCS  ksLRS  Is Mission  J 
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where  UA  V,  GCS ,  LRS,  and  Mission  represent  the  sets  of 
UAVs,  GCSs,  LRSs,  and  missions,  respectively.  The  state 
of  these  individual  components  represent  an  integral  part  of 
the  overall  state  of  the  system. 

The  UAV  state,  sfAVi ,  is  defined  as 


smVl  = 


\W 

>y)t 


tuAvi 

* flight  time, t 
UAVi 

1  recovery  time, t  L 

A  irborne  _  State^A  Vi  =  {atLRS,  airborne } 
sensor  _  package V‘ 

UAV  _typeUAVi 


The  set  accounts  for  the  Cartesian  position,  (x,  y) ,  of  each 

UAV  at  time  t,  the  flight  time  remaining  (if  applicable,  0 
otherwise),  the  recovery  time  remaining  if  located  at  an 
LRS  (if  applicable,  0  otherwise),  whether  the  UAV  is  lo¬ 
cated  at  an  LRS  and  not  airborne  (atLRS)  or  in  flight  (< air- 
born) ,  the  sensors  that  the  UAV  is  carrying  at  time  t ,  and 
the  type  of  UAV  including  all  the  performance  characteris¬ 
tics  associated  with  that  UAV  type.  Performance  character¬ 
istics  for  a  UAV  include 


time  for  a  GCS  is  UAV  dependent  and  is  given  in  the  set 
SyAVi  as  a  UAV  performance  characteristic. 

The  mission  state,  S^isswni ,  is  defined  as 


*  U 

seMissiotij 


/  \  Mission , 

(x>y) 

sensor  _  requirements  s 


where  a  sensor  requirement,  sensor  requirements s,  is  de¬ 
fined  as: 


{sensor  _types 

U  (open  time _availablet  , end  time _availablet  ) 

t\&Time  _  segments  s 

value  rate s 


The  set  accounts  for  the  Cartesian  position,  (x,y) ,  of  each 

mission  which  is  constant  and  the  sensor  requirements  of 
the  mission  to  include  sensor  type,  open  time  segments, 
and  the  value  rate  received  during  the  open  time  segments. 
The  value  rate  is  multiplied  by  the  UAV  time-on-station  if 
the  UAV  has  the  appropriate  sensor  in  its  sensor  package. 

5.2  Control  Space 


•  Speed, 

•  Operating  time, 

•  Transition  time, 

•  Sensor  package, 

•  Available  start  time,  and 

•  Operating  radius. 

The  GCS  state,  sfCSj ,  is  defined  as 


sGCSj 


\Gi 

x>y)  t 


capacity  _  remaining  t 

T  T  A  TT  GCS  j 

UAV  _  type  1 


The  dynamic  programming  formulation  can  be  solved  by 
solving  the  recursive  equations: 

JASt)  =  m^CASt^t)  +  Jt+ASt+r)f°r  t  =  0,...,T -Tf 

uteU 

where  t  f  is  the  time  difference  from  the  end  of  the  time 
horizon  and  the  last  applied  control,  ul_T^ .  Since  all  mis¬ 
sions  are  known  a  priori ,  ideally  the  optimal  control,  u! , 
could  be  obtained  by 


ut  =argma xCt(St,ut)  +  Jt+T(St+T) for  t  =  0,...,T-rf 

uteU 


The  set  accounts  for  the  Cartesian  position,  (x,  y) ,  of  each 

GCS  at  time  t ,  the  capacity  remaining,  and  the  type(s)  of 
UAV(s)  that  the  GCS  can  control.  The  maximum  radius  of 
control  for  a  GCS  is  UAV  dependent  and  is  given  in  the  set 
S^AVi  as  a  UAV  performance  characteristic. 

The  LRS  state,  S^RSk ,  is  defined  as 


g  LRSk 


/  \ LRS ^ 

Vy), 

<  capacity  _  remaining^ k 
UAV  _typeLRSk 


Unfortunately,  because  of  the  combinatorial  nature  of 
the  problem,  this  formulation  suffers  from  the  curse  of  di¬ 
mensionality  in  both  the  state  and  control  space.  Therefore, 
the  cost-to-go  function,  jt(st ),  is  not  easily  obtainable. 
The  constrained  value  optimizer  of  DAFS  can  be  used  in¬ 
stead  to  obtain  an  approximate  solution  using  the  mission 
state  “in  the  near  future.”  This  value  “in  the  near  future”  is 
substituted  for  jt  (st ) .  This  mission  state  will  be  described 
further  in  Section  6. 

5.3  State  Dynamics 


The  set  accounts  for  the  Cartesian  position,  (x,  y ) ,  of  each 

LRS  at  time  t ,  the  capacity  remaining,  and  the  type(s)  of 
UAV(s)  that  the  LRS  can  land  and  recover.  The  recovery 


DAFS  implements  a  discrete  event  world  view  using  event 
graph  methodology  developed  originally  by  Schruben 
(1983).  Event  graph  methodology  assumes  an  event  list 
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which  advance  to  the  next  event  on  the  list  and  can  be  rep¬ 
resented  graphically,  as  shown  in  Figure  1.  Each  node 
represents  a  state  transition  and  each  arc  represents  a 
scheduling  relationship  through  which  an  event  can  cause 
other  events  to  be  placed  on  the  event  list.  In  the  example 
shown,  a  mission  could  be  assigned  to  a  UAV  platform  by 
the  CVO  object  and  that  mission  might  be  out  of  the  range 
of  all  sensors  on  the  platform.  This  event  triggers  a  Start 
Move  event  for  the  platform  in  order  to  move  the  platform 
within  the  range  for  the  sensor.  When  the  Start  Move  event 
occurs,  the  End  Move  event  would  be  placed  on  the  event 
list  with  a  time  delay  of  t,  which  in  this  case  would  be  the 
amount  of  time  it  would  take  the  platform  to  travel  to  its 
destination 


Figure  1:  Event  Graph  Example 


DAFS’  event  graph  methodology  handles  all  of  the 
dynamic  programming  state  dynamics  in  an  efficient  way. 
The  simulation’s  can  easily  represent  the  dynamics,  espe¬ 
cially  of  entities  moving  and  changing  Ground  Control  Sta¬ 
tion  capacities.  For  further  discussion  of  modeling  move¬ 
ment  using  Event  Graph  Methodology,  see  Buss  and 
Sanchez  (2005). 

6  THE  ASSIGNMENT  SCHEDULING 

CAPABILITY  FOR  UAV  SYSTEMS  APPROACH 

As  mentioned  previously,  ASCU  does  not  use  simulation 
in  the  traditional  manner.  Instead,  it  uses  simulation  to 
transition  and  capture  the  state  of  the  system  dynamics,  es¬ 
pecially  the  movement  of  key  entities  (UAVs,  GCSs,  and 
LRSs)  while  applying  controls  using  optimization  in  order 
to  construct  feasible  UAV  schedules. 

ASC-U  periodically  provides  controls  to  available 
UAVs  by  solving  an  assignment  problem  based  on  a  deci¬ 
sion  time  window  into  the  future.  There  are  thus  two  pa¬ 
rameters  of  the  optimization  logic  of  ASC-U:  the  duration 
of  the  decision  time  window  (the  amount  of  future  time 
considered  in  each  decision)  and  the  time  between  re¬ 
optimizations.  These  parameters  can  be  independently  set 
in  the  input  files. 

When  an  optimization  event  occurs,  ASC-U  assigns 
currently  available  UAVs  to  Missions  that  are  open  within 
a  fixed  time  horizon  in  the  future  that  is  dependent  on  the 
type  of  UAV  being  assigned.  This  assignment  is  made  to 
maximize  the  total  value  obtained  by  assigning  the  avail¬ 
able  UAVs  to  Mission  Areas  that  will  have  some  sensor 
demand  in  the  time  horizon.  This  fixed  time  horizon  is  the 


same  for  UAVs  of  the  same  type  but  differs  for  UAVs  of 
differing  types.  The  reason  for  this  difference  is  that  ASC- 
U  considers  each  UAV  type’s  maximum  cycle  time.  A 
UAV’s  maximum  cycle  time  is  its  maximum  endurance 
time  plus  recovery  time.  This  maximum  cycle  time  is  mul¬ 
tiplied  by  a  constant  which  is  determined  experimentally. 

For  example,  consider  2  UAVs  assigned  to  4  missions 
as  depicted  in  Figure  2.  ASC-U  performs  optimal  assign¬ 
ments  at  predetermined  time  intervals  (1).  At  time  to ,  as¬ 
sume  two  UAVs  are  available,  UAV1  and  UAV  2.  The 
tool  considers  all  available  UAVs  with  available  GCS  con¬ 
trol  and  all  missions  with  value  in  the  fixed  time  horizon 
time  window  (missions  1,2,3  but  not  4  for  time  t0)  (2).  A 
UAV  is  assigned  to  a  mission  and  can  service  any  sensor 
requirement  associated  with  that  mission  if  it  has  the  cor¬ 
rect  sensor.  Assume  the  optimal  assignment  is  UAV  1  as¬ 
signed  to  Mission  1  and  UAV  2  assigned  to  Mission  2. 
UAV  1  is  launched  immediately  to  arrive  at  Mission  1  as 
soon  as  possible  (3).  UAV  2  does  not  launch  but  is  sched¬ 
uled  for  launch  to  arrive  just-  in-time  for  the  beginning  of 
Mission  2.  At  time  th  UAVs  that  are  scheduled  to  launch 
but  have  not  yet  launched  are  “unassigned”  and  considered 
for  assignment  (UAV  2  is  unassigned)  (2).  UAV  2  is  again 
assigned  to  Mission  2  and  scheduled  for  launch  to  arrive 
just-in-time.  UAV  assignments  do  not  change  once  they 
have  been  launched.  By  advancing  in  time,  each  UAV  has 
a  schedule  constructed  that  reflects  the  optimal  assignment 
for  available  UAVs  considering  the  fixed  time  horizon  in 
the  future. 


Figure  2:  Assignment  Dynamics  Example 

The  value  of  each  mission  is  determined  by  the  user- 
defined  valuation  rate  of  sensor  requirements  for  each  mis¬ 
sion.  These  user-defined  sensor  valuation  rates  can  depend 
on  the  echelon  requiring  the  mission  or  the  priority  given 
to  the  mission  or  a  combination.  The  UAV  receives  a  total 
value  for  missions  by  determining  the  time- on- station  of 
the  UAV  and  multiplying  it  by  the  value  rate  if  the  sensor 
requirement’s  end  time  is  not  exceeded  or  by  the  value  rate 
from  the  UAV  arrival  to  the  sensor  requirement’s  end  time 
if  the  mission’s  end  time  is  exceeded  by  the  time-on- 
station  of  the  UAV.  These  values  are  then  added  to  deter¬ 
mine  the  possible  value  of  each  mission  if  the  UAV  were 
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assigned  to  the  mission.  Finally,  if  the  mission  is  not  within 
a  GCS  footprint  with  capacity  then  the  value  is  set  to  zero. 

Figure  3  shows  the  different  stages  of  ASC-U  as  it 
constructs  the  UAV  schedules.  Given  a  state  of  the  system 
in  the  lower  left  of  Figure  3,  a  periodically  occurring  opti¬ 
mization  event  is  scheduled  to  occur.  This  optimization 
event  first  uses  a  Value  of  Potential  Assignment  Genera¬ 
tor  to  calculate  the  value  received  within  the  UAV-type 
time  windows  for  every  possible  assignment  for  every 
UAV.  This  value  takes  into  account  movement  of  entities 
and  GCS  range  constraints.  These  assignment  values  along 
with  constraints  on  available  GCS  capacity  are  handed  off 
to  the  optimization.  The  optimization  determines  the 
maximum  value  of  this  optimization  problem  and  hands 
these  assignments  to  the  UAV  Scheduler.  The  UAV 
Scheduler  determines  when  the  UAV  is  launched  so  that 
the  UAV  arrives  just-in-time  if  the  mission  start  time  mi¬ 
nus  the  travel  time  to  the  mission  is  greater  than  the  current 
time.  The  next  optimization  event  is  scheduled  to  take 
place  after  a  delay  of  the  optimization  time  interval.  The 
Dynamic  Simulation  State  Transitions  then  occur  until  that 
event  occurs,  so  that  it  will  generally  be  considering  a  very 
different  state  than  the  previous  optimization. 


•  Let  //  be  the  sub-set  of  all  sensor  packages  cur¬ 
rently  located  at  LRS  l  e  L . 

•  Let  Yga  =  1  if  mission  area  a  is  assigned  to  GCS  g, 
0  otherwise  (by  a  heuristic  discussed  below). 

•  Let  Cja  =  The  reward  for  a  UAV  with  sensor  pack¬ 
age  j  being  assigned  to  mission  area  a  from  the 
soonest  possible  arrival  time  of  the  UAV  at  the 

area  to  the  end  of  the  time  horizon,  t  +  At1 ,  for 
UAV  i. 

•  XJa  =  1  if  a  UAV  with  sensor  package  j  is  assigned 
to  mission  area  a ,  and  0  otherwise. 

Then  the  formulation  is: 


mnxJ_cjaXja 

ja 

(1) 

ZXja  <lVaeA 
j 

(2) 

ZXja  <  1  V/  e  I 

a 

(3) 

ZYgaXja<CgVgeG 

ja 

(4) 

I  XJa<\l,\VUL 

(5) 

jsJ ia 


1 .  Maximize  the  value  of  Mission  Areas  covered. 

2.  Assign  only  1  UAV  per  Mission  Area. 

3.  Assign  only  1  Mission  Area  per  UAV. 

4.  Do  not  exceed  the  GCS  control  limit. 

5.  The  number  of  sensors  assigned  cannot  exceed  the 
number  of  UAVs  available  to  carry  them. 


Figure  3:  ASC-U  Dynamic  Cueing  and  Transitions 

The  following  optimization  problem  is  solved  at  each 
optimization  event. 

•  Let  A  be  the  set  of  all  mission  areas  with  at  least 
one  mission  active  within  the  global  time  horizon. 

•  Let  L  be  the  set  of  all  active  LRS’s 

•  Let  G  be  the  set  of  all  GCS’s. 

•  Let  Gl  the  set  of  all  GCS’s  assigned  to  LRS  L. 

•  Let  Cg  the  number  of  UAV’s  GCS  g  is  capable  of 
controlling. 

•  Let  //  be  the  sub-set  of  all  UAV’s  at  LRS  /  e  L  de¬ 
termined  as  follows:  For  each  LRS  /  e  L,  get  n 
UAV’s  where  n  is  the  minimum  of  the  number  of 
ready  UAV’s,  the  LRS  launch  limit,  the  number 
of  UAV’s  airborne,  the  sum  over  GL  of  Cg,  and 
the  number  of  UAV’s  assigned  to  the  GCS. 


Heuristic  for  determining  (assignment  of  Mission 
Areas’  to  GCS): 

For  each  LRS  and  Mission  Area  a: 

Let  Na  the  number  of  GCS’s  g  that  are  in  range 
of  a  UAV  assigned  to  Mission  Area  a 
For  each  Mission  Area  a ,  sorted  by  Na : 

For  each  GCS  g,  sorted  by  HYga  (so  far) 

a 

If  a  is  in  range  of  g,  let  Yga  =  1. 

Detail  of  determination  of  c iJL 

For  each  UAV  i 

For  each  Mission  Area  a 

Let  tQ  =  the  first  time  after  the  ealiest 
arrival  time  that  UAV  i  can  gain  value 
by  being  assigned  to  Mission  Area  a. 
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Let  tj  =  min(the  latest  time  UAV  i  can 
remain  at  Mission  Area  a ,  the  end  of  t 

he  time  horizon,  t  +  At1 ,  for  UAV  /). 

Let  Ka  =  the  set  of  all  Missions  located 
at  Mission  Area  a. 

Let  Vi;k,to,ti  =  the  value  that  UAV  i  gains 
from  Mission  k  by  being  at  Mis  -sion 
Area  a. 

Then,  cia  =  I  VuKt0A  . 

keKa 

7  INITIAL  RESULTS  AND  CONCLUSIONS 

ASC-U  has  been  shown  in  practice  to  yield  quality  solu¬ 
tions  and  perform  consistently.  This  consistent  perform¬ 
ance  is  characterized  by  increases  in  UAV  performance  re¬ 
sulting  in  corresponding  increases  in  mission  coverage  and 
overall  mission  value.  Furthermore,  a  recent  thesis  by 
Nannini  (2006)  has  also  shown  ASC-U  to  perform  consis¬ 
tently. 

This  paper  demonstrates  the  implementation  of  ap¬ 
proximate  dynamic  programming  techniques  in  a  discrete 
event  simulation  where  simulation  and  optimization  come 
together  doing  what  they  do  best  to  solve  a  very  complex 
scheduling  problem.  The  approach  takes  an  operational 
perspective  so  that  solutions  translate  well  to  implementa¬ 
tion  in  the  real  world  or  to  a  scheduling  approach  within  a 
larger  simulation. 

Simulations  allow  measures  of  performance  to  be  eas¬ 
ily  obtained  from  the  dynamics  and  current  state  of  the 
simulation.  This  capability  allows  for  the  following  meas¬ 
ures  of  performance  to  be  easily  obtained  in  ASC-U: 

•  Sensor  Package  Utilization, 

•  Percent  Mission  coverage  by  Mission  type, 

•  Percent  Mission  coverage  by  Mission  and  UAV 
type, 

•  UAV  utilization  over  time,  and 

•  Ground  Control  Station  load  over  time. 

ASC-U  also  accounts  for  the  effectiveness  of  remote  view¬ 
ing  terminals  (RVT)  which  allow  users  to  see  sensor  data 
without  any  control  of  the  actual  UAV.  Even  though  these 
RVTs  do  not  effect  the  UAV  schedule  solution,  they  are 
easily  inserted  within  the  simulation  tool. 

The  ASC-U  model  uses  concepts  from  both  simulation 
and  optimization.  Rather  than  using  traditional  simulation 
or  optimization  techniques  in  isolation,  it  utilizes  each 
methodology  to  it  advantage.  Simulation  is  used  to  capture 
dynamic  transitions  of  a  sequential  UAV  scheduling  deci¬ 
sion  problem,  and  optimization  is  used  to  allocate  UAVs  to 
mission  areas  within  the  scenario.  ASC-U  thus  exploits 
the  strengths  of  simulation  and  optimization  to  construct 


good,  timely  solutions  that  neither  optimization  or  simula¬ 
tion  could  achieve  alone. 
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